TC-85: tunnel relay MVP -- remote reachability for local nodes (+ TC-244 batch) - #1
Conversation
ws: WebSocket server/client for the tunnel relay (TC-85). @peculiar/x509: replaces node-forge for CSR parsing so ECDSA P-256 CSRs are accepted (TC-244); promoted from a transitive (via acme-client) to a direct dependency.
assertCsrMatchesDomain (names.ts) now parses CSRs with @peculiar/x509
instead of node-forge. Forge cannot parse a CSR with an EC public key
at all ("OID is not RSA"), so every ECDSA node CSR was being rejected
before it even reached the CN/SAN check. @peculiar/x509 reads the
subject/SAN independently of the key algorithm, so RSA and ECDSA CSRs
are now validated identically.
Also updates the test-only CSR/fake-ACME helpers (test-support/csr.ts,
test-support/fake-acme-client.ts) to use @peculiar/x509 so the full
claim -> CSR -> DNS-01 -> certificate round trip is exercised for both
key types in tests, not just the CN/SAN validation in isolation.
Explains why POST /v1/certs/:name persists the bumped sequence before calling the (slow, non-idempotent) ACME issuer rather than after, and the resulting tradeoff (a failed ACME call still consumes the sequence bump; the node must retry with a higher sequence, not the same one).
…uting Adds the remote-reachability tunnel relay: a node opens one outbound WebSocket to wss://<host>/v1/tunnel/<name>, authenticates with the same signed-payload scheme as claim/delete/cert (action: "tunnel", subject must own <name>, sequence anti-replay -- src/names.ts), and the relay then proxies inbound HTTPS requests for <name>.tinycloud.link down that socket as simple JSON request/response frames (src/tunnel/protocol.ts). - src/tunnel/registry.ts: one live socket per name, newest-wins eviction (a fresh registration closes the prior socket with a distinct close code rather than rejecting the new connection). - src/tunnel/proxy.ts: sends request/requestBody frames, demuxes response/responseBody/error frames by request id (concurrency-safe, many in-flight requests can share one socket), with a timeout. - src/tunnel/upgrade.ts: the WS handshake -- auth-frame timeout, name ownership + signature verification, sequence bump (same bump-before-side-effect pattern as the cert flow), ws ping/pong heartbeat with dead-peer termination. - src/tunnel/host-router.ts: Hono middleware that proxies any request whose Host header names a claimed tunnel through its socket, and leaves the control-plane API's own hostname (api.tinycloud.link) untouched. Wired into ServerConfig as an opt-in `tunnelRegistry` field -- omitting it (as all existing callers/tests do) leaves /v1 routing byte-for-byte unchanged. - src/index.ts: constructs the registry and attaches the WS upgrade handler to the same http.Server @hono/node-server returns. Node-side (Rust client) implementation is out of scope; the wire protocol in src/tunnel/protocol.ts and the auth/lifecycle rules in src/tunnel/upgrade.ts are the contract for it (README documents both precisely). Ingress/TLS wiring for *.tinycloud.link traffic is a separate, deliberately unresolved question here -- see the README's "Ingress and TLS for tunnels" section and the follow-up docker-compose change, neither of which is deployed by this PR.
README: full tunnel relay section -- lifecycle diagram, auth record and close codes, the frame protocol table with the exact rules a Rust node client must follow (src/tunnel/protocol.ts is the source of truth), and the ingress/TLS finding: dstack-ingress terminates all of :443 inside the CVM with certs it obtained itself, so per-name relay certificates obtained by this process could never be served -- the shipped design is one wildcard cert (DOMAIN=*.tinycloud.link on dstack-ingress, native wildcard DNS-01 support) plus Host-header routing in this process, with the dstack-gateway wildcard-TXT support flagged as the open item to verify on a staging CVM before deploying. docker-compose.phala.yml: switch dstack-ingress DOMAIN to the wildcard (constraint + fallback documented inline; NOT deployed by this change) and pass API_HOSTNAME through. .env.example documents API_HOSTNAME.
…te limiting, ordered-pair headers Addresses the round-1 review of the tunnel relay (PR #1): 1. Body/frame size limits: proxied request bodies are now read with a streaming cap (host-router.ts's readLimitedBody, 413 past the cap) and response bodies accumulated in proxy.ts are capped too (502 + an `error` frame back to the node past the cap). Default 25MB, overridable via TUNNEL_MAX_BODY_BYTES. The relay's WebSocketServer now sets `maxPayload` (1MB, protocol.ts's MAX_FRAME_PAYLOAD_BYTES) so no single WS frame can be unbounded; request bodies are chunked into <=256KB requestBody frames (protocol.ts's BODY_CHUNK_BYTES) to respect it. Also adds a 'ws' `error` listener in upgrade.ts -- without one, a maxPayload violation's unhandled 'error' event would crash the whole process, not just the one tunnel. 2. Rate/connection limiting on the WS upgrade path (upgrade.ts, new rate-limit.ts): a per-IP connection-attempt limiter (30/min), a per-name churn limiter (10/min), and a global concurrent-tunnel cap (1000, env TUNNEL_MAX_CONCURRENT) all run in the raw HTTP `upgrade` handler, before the WS handshake and before authenticate()'s nameStore.get Postgres read. Deliberately in-memory (not the Postgres-backed cert rate limiter): these are short, per-minute windows checked on every upgrade attempt. 3. Frame headers are now ordered `[name, value]` pairs (protocol.ts's TunnelRequestFrame/TunnelResponseFrame) instead of a Record, so duplicate header names -- most importantly Set-Cookie -- survive the roundtrip instead of colliding on one object key. host-router.ts rebuilds the final client-facing Response via repeated Headers.append() calls so multiple Set-Cookie entries are preserved end to end. Tests: 9 new (66 -> 75) covering each fix -- request/response body caps, multi-chunk body reassembly, WS maxPayload enforcement, per-IP/per-name/ global connection limits, header-pair passthrough, and a two-Set-Cookie roundtrip.
- Sequence coordination is per-name, not per-operation: claim/delete/cert/ tunnel-auth all share one counter, single source of truth on the node. - Max frame (1MB) and body (25MB, TUNNEL_MAX_BODY_BYTES) size limits, and the requestBody/responseBody chunking contract this implies. - Frames carrying an unrecognized request id are silently ignored. - Headers are ordered [name, value] pairs so duplicate header names (e.g. Set-Cookie) survive. - Connection-attempt limits enforced before the WS handshake (per-IP, per-name churn, global concurrent-tunnel cap via TUNNEL_MAX_CONCURRENT).
Review round complete: security-review fixes appliedAddressed all four items from the round-1 security review, pushed as two commits ( 1. Body/frame size limits — 2. Rate/connection limiting — 3. Headers as ordered pairs — 4. README — documented shared per-name sequence coordination across claim/cert/delete/tunnel, max frame/body sizes and the chunking contract, unknown-id frames being ignored, and the pair-array header format. Validation: |
What
TC-85 (tunnel relay MVP): nodes open one outbound WebSocket to
wss://api.tinycloud.link/v1/tunnel/<name>, authenticate with the same signed-payload scheme as every other write (action: "tunnel", subject must own the name, strictly-increasing sequence), and the relay proxies HTTPS requests forhttps://<name>.tinycloud.linkdown the socket as JSON request/response frames.src/tunnel/protocol.ts— HTTP-over-WS framing (request/requestBody/response/responseBody/error, base64 bodies, multiplexed by request id). Source of truth for the Rust node client; the README documents the full contract (frames, auth, close codes, lifecycle).src/tunnel/upgrade.ts— WS handshake: 5s auth timeout, ownership + signature verification, sequence bump before going live (same bump-before-side-effect pattern as the cert flow), ping/pong heartbeat with dead-peer termination.src/tunnel/registry.ts— one live socket per name, newest wins (a reconnecting node evicts its own stale socket, close code 4410).src/tunnel/host-router.ts— Host-header routing middleware:API_HOSTNAME(defaultapi.tinycloud.link) gets the normal/v1API, any other<name>.tinycloud.linkhost is proxied through its tunnel (502 no tunnel / 504 timeout). Opt-in viaServerConfig.tunnelRegistry; existing behavior is unchanged when unset.TC-244 batch:
assertCsrMatchesDomainparses with@peculiar/x509instead of node-forge (forge cannot parse EC CSRs at all — "OID is not RSA"). RSA still works; both are tested end-to-end through the fake ACME client.Ingress/TLS finding (design honesty)
Per-name relay certificates are a dead end in the Phala deployment model:
dstack-ingress(HAProxy L4) terminates all of the CVM's:443with certificates it obtained itself at bootstrap — this process never sees the TLS handshake and has no way to serve certs it might obtain. Shipped design instead: one wildcard cert viaDOMAIN=*.tinycloud.linkon dstack-ingress (native wildcard DNS-01 support) + Host-header routing behind it. Open item before deploy: dstack-ingress's wildcard mode requires dstack-gateway wildcard TXT resolution (dstack#545) — must be verified on a staging CVM; fallback isDOMAIN=api.tinycloud.link(API unaffected, tunnels dark). Details in README "Ingress and TLS for tunnels".Not in this PR
Validation
tsc --noEmitclean